Web Search Gemo-Lamsade

With a constantly increasing size of billions of freely accessible documents, one of the major issues raised by the World Wide Web is that of searching in an effective and efficient way through these documents to find these that best suit a user's need. The purpose of the chapter is to describe the techniques that are at the core of today's search engines (such as Google 1 , Yahoo! 2 , Microsoft Live Search 3 or Exalead 4), that is, mostly keyword search in collections of text documents. We also briefly touch upon other techniques and research issues that may be of importance in next-generation search engines. This chapter is organized as follows: in Section 1, we discuss the Web and the languages and protocols it relies upon. We then present in Section 2 the techniques that can be used to retrieve pages from the Web, that is, to crawl it. First-generation search engines, exemplified by Altavista 5 relied mostly on the classical information retrieval (IR) techniques that are described in Section 3. With the advent of Google, other techniques that make use of the graph structure of the Web (see Section 4) have very effectively complemented text IR. We then proceed to a brief discussion of currently active research topics about the Web in Section 5. Whereas the Internet is a physical network of computers (or hosts) connected to each other from all around the world, the World Wide Web, WWW or Web in short, is a logical collection of hyperlinked documents shared by the hosts of this network. An hyperlinked document is just a document with references to other documents of the same collection. Note that documents of the Web may both refer to static documents stored on the hard drive of some host of the Internet, and to dynamic documents that are generated on the fly when accessing the document. This means that there is a virtually unlimited number of documents on the Web, since dynamic documents can change on each request. When one speaks of the Web, it is mostly about the public part of the Web, which is freely accessible, but there are also various private Webs that are restricted to some community or company, either on private Intranets or on the Internet, with password-protected pages. Documents, and, more generally, resources on the Web, are identified by a URL (Uniform Resource Locator) which is a …